Crowdsourcing Relevance: Two Studies on Assessment

Crowdsourcing has become an alternative approach to collect relevance judgments at large scale. In this thesis, we focus on some specific aspects related to time, scale, and agreement. First, we address the issue of the time factor in gathering relevance label: we study how much time the judges need to assess documents. We conduct a series of four experiments which unexpectedly reveal us how introducing time limitations leads to benefits in terms of the quality of the results. Furthermore, we discuss strategies aimed to determine the right amount of time to make available to the workers for the relevance assessment, in order to both guarantee the high quality of the gathered results and the saving of the valuable resources of time and money. Then we explore the application of magnitude estimation, a psychophysical scaling technique for the measurement of sensation, for relevance assessment. We conduct a large-scale user study across 18 TREC topics, collecting more than 50,000 magnitude estimation judgments, which result to be overall rank-aligned with ordinal judgments made by expert relevance assessors. We discuss the benefits, the reliability of the judgements collected, and the competitiveness in terms of assessor cost. We also report some preliminary results on the agreement among judges. Often, the results of crowdsourcing experiments are affected by noise, that can be ascribed to lack of agreement among workers. This aspect should be considered as it can affect the reliability of the gathered relevance labels, as well as the overall repeatability of the experiments.openDottorato di ricerca in Informatica e scienze matematiche e fisicheopenMaddalena, Edd

Multidimensional news quality: A comparison of crowdsourcing and nichesourcing

In the age of fake news and of filter bubbles, assessing the quality of information is a compelling issue: it is important for users to understand the quality of the information they consume online. We report on our experiment aimed at understanding if workers from the crowd can be a suitable alternative to

Preliminary results from a crowdsourcing experiment in immunohistochemistry

Background: Crowdsourcing, i.e., the outsourcing of tasks typically performed by a few experts to a large crowd as an open call, has been shown to be reasonably effective in many cases, like Wikipedia, the Chess match of Kasparov against the world in 1999, and several others. The aim of the present paper is to describe the setup of an experimentation of crowdsourcing techniques applied to the quantification of immunohistochemistry. Methods: Fourteen Images from MIB1-stained breast specimens were first manually counted by a pathologist, then submitted to a crowdsourcing platform through a specifically developed application. 10 positivity evaluations for each image have been collected and summarized using their median. The positivity values have been then compared to the gold standard provided by the pathologist by means of Spearman correlation. Results: Contributors were in total 28, and evaluated 4.64 images each on average. Spearman correlation between gold and crowdsourced positivity percentages is 0.946 (p < 0.001). Conclusions: Aim of the experiment was to understand how to use crowdsourcing for an image analysis task that is currently time-consuming when done by human experts. Crowdsourced work can be used in various ways, in particular statistically agregating data to reduce identification errors. However, in this preliminary experimentation we just considered the most basic indicator, that is the median positivity percentage, which provided overall good results. This method might be more aimed to research than routine: when a large number of images are in need of ad-hoc evaluation, crowdsourcing may represent a quick answer to the need. \ua9 Della Mea et al

Observation of the B0 → ρ0ρ0 decay from an amplitude analysis of B0 → (π+π−)(π+π−) decays

Proton–proton collision data recorded in 2011 and 2012 by the LHCb experiment, corresponding to an integrated luminosity of 3.0 fb−1 , are analysed to search for the charmless B0→ρ0ρ0 decay. More than 600 B0→(π+π−)(π+π−) signal decays are selected and used to perform an amplitude analysis, under the assumption of no CP violation in the decay, from which the B0→ρ0ρ0 decay is observed for the first time with 7.1 standard deviations significance. The fraction of B0→ρ0ρ0 decays yielding a longitudinally polarised final state is measured to be fL=0.745−0.058+0.048(stat)±0.034(syst) . The B0→ρ0ρ0 branching fraction, using the B0→ϕK⁎(892)0 decay as reference, is also reported as B(B0→ρ0ρ0)=(0.94±0.17(stat)±0.09(syst)±0.06(BF))×10−6

Angular analysis of the B-0 -&gt; K*(0) e(+) e(-) decay in the low-q(2) region

An angular analysis of the $B^0 \rightarrow K^{*0} e^+ e^-$ decay is performed using a data sample, corresponding to an integrated luminosity of 3.0 {\mbox{fb}^{-1}}, collected by the LHCb experiment in $pp$ collisions at centre-of-mass energies of 7 and 8 TeV during 2011 and 2012. For the first time several observables are measured in the dielectron mass squared ($q^2$) interval between 0.002 and 1.120${\mathrm{\,Ge\kern -0.1em V^2\!/}c^4}$. The angular observables $F_{\mathrm{L}}$ and $A_{\mathrm{T}}^{\mathrm{Re}}$ which are related to the $K^{*0}$ polarisation and to the lepton forward-backward asymmetry, are measured to be $F_{\mathrm{L}}= 0.16 \pm 0.06 \pm0.03$ and $A_{\mathrm{T}}^{\mathrm{Re}} = 0.10 \pm 0.18 \pm 0.05$, where the first uncertainty is statistical and the second systematic. The angular observables $A_{\mathrm{T}}^{(2)}$ and $A_{\mathrm{T}}^{\mathrm{Im}}$ which are sensitive to the photon polarisation in this $q^2$ range, are found to be $A_{\mathrm{T}}^{(2)} = -0.23 \pm 0.23 \pm 0.05$ and $A_{\mathrm{T}}^{\mathrm{Im}} =0.14 \pm 0.22 \pm 0.05$. The results are consistent with Standard Model predictions.An angular analysis of the B$^{0}$ → K^{*}^{0} e$^{+}$ e$^{−}$ decay is performed using a data sample, corresponding to an integrated luminosity of 3.0 fb$^{−1}$, collected by the LHCb experiment in pp collisions at centre-of-mass energies of 7 and 8 TeV during 2011 and 2012. For the first time several observables are measured in the dielectron mass squared (q$^{2}$) interval between 0.002 and 1.120 GeV$^{2}$ /c$^{4}$. The angular observables F$_{L}$ and A$_{T}^{Re}$ which are related to the K^{*}^{0} polarisation and to the lepton forward-backward asymmetry, are measured to be F$_{L}$ = 0.16 ± 0.06 ± 0.03 and A$_{T}^{Re}$  = 0.10 ± 0.18 ± 0.05, where the first uncertainty is statistical and the second systematic. The angular observables A$_{T}^{(2)}$ and A$_{T}^{Im}$ which are sensitive to the photon polarisation in this q$^{2}$ range, are found to be A$_{T}^{(2)}$  = − 0.23 ± 0.23 ± 0.05 and A$_{T}^{Im}$  = 0.14 ± 0.22 ± 0.05. The results are consistent with Standard Model predictions.An angular analysis of the $B^0 \rightarrow K^{*0} e^+ e^-$ decay is performed using a data sample, corresponding to an integrated luminosity of 3.0 {\mbox{fb}^{-1}}, collected by the LHCb experiment in $pp$ collisions at centre-of-mass energies of 7 and 8 TeV during 2011 and 2012. For the first time several observables are measured in the dielectron mass squared ($q^2$) interval between 0.002 and 1.120${\mathrm{\,Ge\kern -0.1em V^2\!/}c^4}$. The angular observables $F_{\mathrm{L}}$ and $A_{\mathrm{T}}^{\mathrm{Re}}$ which are related to the $K^{*0}$ polarisation and to the lepton forward-backward asymmetry, are measured to be $F_{\mathrm{L}}= 0.16 \pm 0.06 \pm0.03$ and $A_{\mathrm{T}}^{\mathrm{Re}} = 0.10 \pm 0.18 \pm 0.05$, where the first uncertainty is statistical and the second systematic. The angular observables $A_{\mathrm{T}}^{(2)}$ and $A_{\mathrm{T}}^{\mathrm{Im}}$ which are sensitive to the photon polarisation in this $q^2$ range, are found to be $A_{\mathrm{T}}^{(2)} = -0.23 \pm 0.23 \pm 0.05$ and $A_{\mathrm{T}}^{\mathrm{Im}} =0.14 \pm 0.22 \pm 0.05$. The results are consistent with Standard Model predictions

A study of CP violation in B-+/- -&gt; DK +/- and B-+/- -&gt; D pi(+/-) decays with D -&gt; (KSK +/-)-K-0 pi(-/+) final states

A first study of CP violation in the decay modes $B^\pm\to [K^0_{\rm S} K^\pm \pi^\mp]_D h^\pm$ and $B^\pm\to [K^0_{\rm S} K^\mp \pi^\pm]_D h^\pm$, where $h$ labels a $K$ or $\pi$ meson and $D$ labels a $D^0$ or $\overline{D}^0$ meson, is performed. The analysis uses the LHCb data set collected in $pp$ collisions, corresponding to an integrated luminosity of 3 fb$^{-1}$. The analysis is sensitive to the CP-violating CKM phase $\gamma$ through seven observables: one charge asymmetry in each of the four modes and three ratios of the charge-integrated yields. The results are consistent with measurements of $\gamma$ using other decay modes

Study of the rare B-s(0) and B-0 decays into the pi(+) pi(-) mu(+) mu(-) final state

A search for the rare decays $B_s^0 \to \pi^+\pi^-\mu^+\mu^-$ and $B^0 \to \pi^+\pi^-\mu^+\mu^-$ is performed in a data set corresponding to an integrated luminosity of 3.0 fb$^{-1}$ collected by the LHCb detector in proton-proton collisions at centre-of-mass energies of 7 and 8 TeV. Decay candidates with pion pairs that have invariant mass in the range 0.5-1.3 GeV/$c^2$ and with muon pairs that do not originate from a resonance are considered. The first observation of the decay $B_s^0 \to \pi^+\pi^-\mu^+\mu^-$ and the first evidence of the decay $B^0 \to \pi^+\pi^-\mu^+\mu^-$ are obtained and the branching fractions are measured to be $\mathcal{B}(B_s^0 \to \pi^+\pi^-\mu^+\mu^-)=(8.6\pm 1.5\,({\rm stat}) \pm 0.7\,({\rm syst})\pm 0.7\,({\rm norm}))\times 10^{-8}$ and $\mathcal{B}(B^0 \to \pi^+\pi^-\mu^+\mu^-)=(2.11\pm 0.51\,({\rm stat}) \pm 0.15\,({\rm syst})\pm 0.16\,({\rm norm}) )\times 10^{-8}$, where the third uncertainty is due to the branching fraction of the decay $B^0\to J/\psi(\to \mu^+\mu^-)K^*(890)^0(\to K^+\pi^-)$, used as a normalisation.A search for the rare decays Bs0→π+π−μ+μ− and B0→π+π−μ+μ− is performed in a data set corresponding to an integrated luminosity of 3.0 fb−1 collected by the LHCb detector in proton–proton collisions at centre-of-mass energies of 7 and 8 TeV . Decay candidates with pion pairs that have invariant mass in the range 0.5–1.3 GeV/c2 and with muon pairs that do not originate from a resonance are considered. The first observation of the decay Bs0→π+π−μ+μ− and the first evidence of the decay B0→π+π−μ+μ− are obtained and the branching fractions, restricted to the dipion-mass range considered, are measured to be B(Bs0→π+π−μ+μ−)=(8.6±1.5 (stat)±0.7 (syst)±0.7(norm))×10−8 and B(B0→π+π−μ+μ−)=(2.11±0.51(stat)±0.15(syst)±0.16(norm))×10−8 , where the third uncertainty is due to the branching fraction of the decay B0→J/ψ(→μ+μ−)K⁎(892)0(→K+π−) , used as a normalisation.A search for the rare decays Bs0→π+π−μ+μ− and B0→π+π−μ+μ− is performed in a data set corresponding to an integrated luminosity of 3.0 fb−1 collected by the LHCb detector in proton–proton collisions at centre-of-mass energies of 7 and 8 TeV . Decay candidates with pion pairs that have invariant mass in the range 0.5–1.3 GeV/c2 and with muon pairs that do not originate from a resonance are considered. The first observation of the decay Bs0→π+π−μ+μ− and the first evidence of the decay B0→π+π−μ+μ− are obtained and the branching fractions, restricted to the dipion-mass range considered, are measured to be B(Bs0→π+π−μ+μ−)=(8.6±1.5 (stat)±0.7 (syst)±0.7(norm))×10−8 and B(B0→π+π−μ+μ−)=(2.11±0.51(stat)±0.15(syst)±0.16(norm))×10−8 , where the third uncertainty is due to the branching fraction of the decay B0→J/ψ(→μ+μ−)K⁎(892)0(→K+π−) , used as a normalisation.A search for the rare decays $B_s^0 \to \pi^+\pi^-\mu^+\mu^-$ and $B^0 \to \pi^+\pi^-\mu^+\mu^-$ is performed in a data set corresponding to an integrated luminosity of 3.0 fb$^{-1}$ collected by the LHCb detector in proton-proton collisions at centre-of-mass energies of 7 and 8 TeV. Decay candidates with pion pairs that have invariant mass in the range 0.5-1.3 GeV/$c^2$ and with muon pairs that do not originate from a resonance are considered. The first observation of the decay $B_s^0 \to \pi^+\pi^-\mu^+\mu^-$ and the first evidence of the decay $B^0 \to \pi^+\pi^-\mu^+\mu^-$ are obtained and the branching fractions, restricted to the dipion-mass range considered, are measured to be $\mathcal{B}(B_s^0 \to \pi^+\pi^-\mu^+\mu^-)=(8.6\pm 1.5\,({\rm stat}) \pm 0.7\,({\rm syst})\pm 0.7\,({\rm norm}))\times 10^{-8}$ and $\mathcal{B}(B^0 \to \pi^+\pi^-\mu^+\mu^-)=(2.11\pm 0.51\,({\rm stat}) \pm 0.15\,({\rm syst})\pm 0.16\,({\rm norm}) )\times 10^{-8}$, where the third uncertainty is due to the branching fraction of the decay $B^0\to J/\psi(\to \mu^+\mu^-)K^*(890)^0(\to K^+\pi^-)$, used as a normalisation

Measurement of the Z plus b-jet cross-section in pp collisions at root s=7 TeV in the forward region

The associated production of a Z boson or an off-shell photon $\gamma^*$ with a bottom quark in the forward region is studied using proton-proton collisions at a centre-of-mass energy of $7{\mathrm{\,Te\kern -0.1em V}}$. The Z bosons are reconstructed in the ${\text{Z}/\gamma^*}\rightarrow{\mu^{+}\mu^{-}}$ final state from muons with a transverse momentum larger than $20{\mathrm{\,Ge\kern -0.1em V}}$, while two transverse momentum thresholds are considered for jets ($10{\mathrm{\,Ge\kern -0.1em V}}$ and $20{\mathrm{\,Ge\kern -0.1em V}}$). Both muons and jets are reconstructed in the pseudorapidity range $2.0 10{\mathrm{\,Ge\kern -0.1em V}}$, and \sigma(\text{\text{Z}/\gamma^*(\mu^{+}\mu^{-})+b-jet}) = 167 \pm 47 (\text{stat}) \pm 29 (\text{syst}) \pm 6 (\text{lumi}) {\,{fb}} for {p_{\rm T}}(jet)$>20{\mathrm{\,Ge\kern -0.1em V}}$